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ABSTRACT 



Results-based evaluation and assessment of guidance and 



counseling is a difficult task. How to best evaluate whether the 
comprehensive guidance program was getting the intended results became an 
issue for Omaha Public Schools. They set about to draft a framework for 
measuring student behavior or learning resulting from activities associated 
with the four major components of comprehensive guidance (guidance 
curriculum; indirect planning; responsive services; system support) . The 
evaluation framework included process -based, results-based, and 
data-gathering document components. Counselors from all grade levels were 
represented on the working committee. District administrators from the 
Department of Research and the Department of Curriculum and Instruction were 
also involved. Everyone, including teachers, parents, and students, had some 
input into the evaluation. Recommendations for pursuing results-based 
evaluation, as well as lessons learned through this evaluation, are included. 
(Contains 2 figures and 30 references.) (JDM) 
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Chapter Thirteen 



Comprehensive Guidance 
Results-Based Evaluation: 

Developing a Practical Approach 

Stan J. Maliszewski 

The University of Arizona, Tucson, Arizona 
& 

John J. Mackiel 

Omaha Public Schools, Omaha, Nebraska 

“Don’t report only what school counselors are doing. Report how students 
are different because of counselors delivering the guidance program.” 
This was the charge of the superintendent to counselors in the Omaha 
Public Scl^ools in 1987. Today the current superintendent of the Omaha 
Public Schools, school staff, and the Omaha community ask questions 
regarding results reporting and guidance plan implementation. Perhaps 
the questions are reframed, but the current school superintendent and 
stakeholders in the guidance program are still focused on “results-based 
evaluation.” With greater restrictions on public spending, the need to 
validate the use of school counselors and ensure the growth of the school 
guidance program is more apparent than ever before (Gillies, 1993). 

While district leaders and others are asking questions about guidance 
accountability and evaluation, there is a growing interest in guidance as a 
program rather than simply a service. Services are often ancillary and are more 
vulnerable during difficult budget decisions. Numerous state departments of 
education have adopted comprehensive competency-based guidance as the 
model for delivering a guidance program to all students. Comprehensive 
competency-based guidance results in a well-defined proactive and reactive 
program (Starr & Gysbers, 1988). 

A Comprehensive Competency-Based Approach to Guidance 

Counselors in the Omaha Public Schools realize that an inquiry as to 
what counselors do provides an opportunity to educate others about their 
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school’s comprehensive guidance program (Johnson & Johnson, 1991). 
A well-defined K— 12 comprehensive school guidance program consists 
of four major components (guidance curriculum, individual pla nnin gs 
responsive services, and system support) delivered in three domains 
(career/occupational, academic/educational, personal/social) to all 
students while the unique needs of individual students are simultaneously 
addressed (Maliszewski, Pilkington, & Radd, 1994-95). 



Comprehensive competency-based guidance is an integral part of the 
Omaha Public Schools total educational program. The program is organized 
and implemented by school counselors with the support of teachers, 
administrators, community agencies, and advisory committees. To this extent, 
it is indeed a school and community collaborative effort (Maliszewski, 
Pilkington, & Radd, 1996). 

Results-based evaluation and assessment of guidance and counseling have 
proven to be the most difficult tasks encountered since the district adopted 
comprehensive guidance (Moore, 1999). Answering the question, “How are 
students different as a result of the school guidance program?” continues to be 
a challenge for district counselors. The good news is that progress is being 
made. Counselors are approaching the challenge with creative K-12 pilot 
evaluation projects that are getting closer to capturing the results of the 
comprehensi^^ guidance program. 

Determining a Method of Program Evaluation 

The first step toward determining a method of program evaluation and 
counselor accountability for the district was the organization of a working 
committee to review research and to identify other school districts that 
were attempting to conduct results-based evaluation. The supervisor of 
guidance chaired this committee. The 18 members consisted of 
counselors from all grade levels, university counselor educators, and 
district-level administrators representing curriculum and instruction, 
research, psychological services, and connectivity. A relatively large 
committee was selected because of the extent of work involved and as a 
means to gamer districtwide support. 

Committee members were enthusiastic about the charge and continued to 
maintain high interest. The agenda for the first meeting consisted of a review of 
the charge, identification of the purpose of assessment (“to provide school 
counselors with evidence of their contributions” and “to focus efforts on 
improving an activity or component of the comprehensive guidance program”). 
Also discussed was the manner in which the study would proceed. 
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Learning about Evaluation 

A subcommittee was formed to conduct a review of literature related to 
guidance program evaluation. Another subcommittee was charged with 
contacting school districts that were attempting to evaluate their guidance 
programs. These school districts were identified from publications, 
professional networking, and professional organizations (e.g., the 
American School Counselor Association and the College Board). 

A report outlining the results of the literature review was presented at the 
second meeting. Among the resources identified as referencing guidance 
evaluation were 58 journal articles, an ERIC/CASS Special Digest Collection 
(Schafer, 1995), a special issue on research and evaluation in school counseling 
published by the American Counseling Association (Gerler, 1990), and a 
practitioner’s guide to evaluating guidance programs (Johnson & Whitfield, 
1991). Each resource was examined for practicality of implementation! 
Committee members came to the consensus that although each publication was 
well researched and reported, none had a methodology congruent with the culture 
and common practices of the guidance program in the Omaha Public Schools. 
Several articles did contribute to determining a direction for evaluation, however 
(Bloom,1994a, 1994b; Gysbers, Hughey, Starr, & Lapan, 1992; Hughey, 
Gysbers, & Starr, 1993). Moreover, in recent years, published manuscripts have 
contained meAodology and results that will move our profession closer to 
determining changes in student behavior and retention of delivered guidance 
competencies (Burnham & Jackson, 2000; Kaplan, 2000; Lapan, Gysbers, & 
Sun, 1999; Schlossberg, Morris, & Lieberman, 2001 ; Trevisan & Hubert, 2001 ; 
Zmck & Littrell, 2001). Finally, Henderson and Gysbers (1998) offer a concise 
description and practical application of comprehensive guidance program 
concepts in selected school districts across the country. 

Some school districts were particularly helpful in this study of evaluation 
and have remained in a networking partnership. School district representatives 
exchanged information and learned from each other. Members of the network 
are the departments of school guidance and counseling in Corpus Christi School 
District, Des Moines Independent Community School District, Mesa Public 
Schools, Milwaukee Public Schools, and Tucson Unified School District, as 
well as the Arizona, Missouri, Ohio, and Utah State Departments of Education. 
All of these school districts have conducted some means of results-based 
evaluation and are exchanging information with each other. The sharing of 
materials, procedures, pilot studies, and documented results remains a vital 

aspect m determining how our profession can conduct practical, results-based 
evaluation. 
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Creating a Framework for Evaluation 

The literature review and networking with other school districts offered 
the committee a direction for their work. The committee’s next goal 
was to draft a framework for measuring student behavior or learning 
resulting from activities associated with each of the four major 
components of comprehensive guidance. This framework served as a 
working document allowing everyone to see the direction for evaluation. 
Figure 1 depicts the framework used to ensure counselors focused on 
student results, made distinctions in evaluation, and reported 
documentation. 



Figure 1. Framework for Evaluation 



Type of Evaluation 




Component 




Process Based 




Results Based 




Data-Gathering Document 




I. Guidance Curriculum 




II. Individual Planning 


\ • 


ni. Response Services 




rv. Systeip Support 




V. Program Audit 





The initial recommendation to address one component each year and 
conduct pilot approaches to assess student and program results for that 
component proved to be too ambitious. This was true especially for the 
intermediate evaluation of student learning associated with delivering the 
guidance curriculum. It was determined that a minimum of three years, perhaps 
more, was needed for modifying intermediate evaluation measurements and 
piloting refinements to the measurement and gathering of documentation 
(Moore, 1999). 

Addressing Process-Based and Results-Based Evaluation 

Although results-based evaluation was the primary goal, it was decided 
that process-based evaluation would also be addressed. The committee 
hoped that using both forms of evaluation would result in a more 
comprehensive, and perhaps more practical, means of accountability 
until more could be learned about how to measure guidance program 
effects on students. In addition, presenting the distinction between 
process-based and results-based evaluation was a means of educating 
counselors relative to the primary focus — changes in student behavior 
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and learning (Molyneaux, 1999). All Omaha Public Schools had reported 
conducting some form of process-based evaluation resulting in 
improvements to the school guidance and counseling programs. These 
process-based measures included the following: 

• completion of teaching and delivery guidance curriculum competencies; 

• number of guidance lessons/activities; 

• increase in the number of individual counseling sessions; 

• completion of a career assessment by all students in each grade level 
(7-12); 

• increase in student participation in college admissions test preparation 
programs; 

• increase in the number of small groups offered and student participation 
in them; 

• increase in the number of parent conferences, individual student 
conferences, home visits, agency contacts, staff conferences, chemical 
abuse reports, and staff development in-services/presentations; and 

• decrease in clerical responsibilities and time devoted to supervision. 
These process-based measures indicated apparent improvements in the 

guidance program, but a means of evaluating actual results remained the primary 
target. Results-based evaluation demands constant attention. Several schools 
piloted intermediate results-based evaluation. A sampling of measurements used 
thus far to determine effectiveness of the comprehensive guidance program 
and its effects on students include the following: 

• evaluation of students’ mastery of specified competencies immediately 
after a guidance lesson is taught (short-term results); 

• benchmark evaluation of guidance curriculum competencies using a 
sample (15%) of students and two competencies in each domain for 
each K— 12 grade level; 

• creation of a written career and advanced education plan for each 
graduating student; 

• decrease in students dropping out and not attending alternative 
educational programs; 

• decrease in suspensions and office referrals in those schools (K-12) 
that have implemented conflict management programs; 

• decrease in the number of students who request schedule changes; 
increase in reports of pre-existing child abuse in schools that have 
elementary school counselors; 

• increase in self-esteem as measured by the Piers-Harris Instrument in 
elementary schools with counselors; 

student, parent, and staff attitudinal surveys administered in elementary 
schools showing 96% to 98% positive responses to survey items; and 
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• increase in SAT I and ACT participation and a slight increase in scores 
of students who participated in a college admissions test preparation 
program. 

The Missouri Model Audit (Gysbers & Henderson, 1994) was used to 
verify that the same guidance program was delivered in all schools throughout 
the district. Recently, an audit developed by the Center for Educational 
Development and the Arizona counselor leadership cadre offers a concise means 
of monitoring state standards for comprehensive competency-based programs 
(Lawhead, 2001). A complete description of the evaluation methods and 
recommended data-gathering documentation can be found in the Guide for 
Comprehensive Guidance Program Evaluation and Documents (Maliszewski 
1994). 

District Involvement and Support 

Ensuring that counselors from all grade levels were represented on the 
working committee proved to be a crucial factor in studying results- 
based evaluation. However, equally valuable was the involvement of 
district administrators representing the Department of Research and 
Department of Curriculum and Instruction. When discussing how to 
evaluate student learning resulting from the guidance* curriculum, the 
coordinator of curriculum and instruction recommended prioritizing two 
competencies in each domain (personal/social, career/occupational, and 
academic/educational) for results-based evaluation at each grade level 
from K to 12. Advisory committee members, students, parents, 
counselors, teachers, and administrators all provided input regarding 
which two domain competencies should be assessed at each grade level. 
(The curriculum contains more than 1 00 identified competencies at each 
grade level.) 

Computer technology for gathering data, scanning equipment, and 
materials were made available through the support and cooperation of the 
Research Department. Another contribution came from the research director, 
who suggested that intermediate evaluation of a random sample of 15% of all 
students receiving guidance instruction should be undertaken to demonstrate 
retention of learning three to five months after the targeted competencies were 
taught. Furthermore, it was recommended that 80% be set as the criterion for 
demonstrating mastery. During the first year of piloting this intermediate 
evaluation, between 47% and 97% of the students in the various schools 
demonstrated mastery. 

Given the high mobility of students in the district, especially at the 
elementary level, it became apparent that counselors had to agree on specific 
months when targeted competencies would be delivered and assessed in all the 
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schools, so that all students throughout the district would leam the same grade- 
level competencies within the same time frame. Thus, students who transferred 
between schools within the district could be included in the sample population 
(Luther, 2001). 

Obstacles to results-based evaluation expressed by counselors included 
lack of time, inadequate training, lack of support, shortage of resources, and 
concern about how results would be reported and used (Paisley & Hayes, 1 997). 
District administrative support staff helped to educate counselors about 
evaluation and supported data gathering efforts. Through a reallocation of 
resources and a streamlining of other responsibilities, one counselor-project 
assistant ultimately was given the additional duty of assisting with districtwide 
guidance evaluation. The project assistant was assigned to work out of the district 
office but devote the majority of time to schools. The assistant’s duties included 
monitoring the assessment process and gathering and reporting all data. (The 
framework for gathering data is shown in Figure 2.) Efforts were directed at 
determining results of the delivery of the guidance curriculum competencies, 
but other components were also assessed during the year. Although each school 
had a choice of conducting its own evaluation and forwarding the results rather 
than being evaluated by the project assistant, most counselors, were quite relieved 
that the evaluation process did not add to their already busy schedules. 

Figure 2. i^ramework for Guiding Assessment of Competencies, 
Student Results, and Gathering of Data (Luther, 2001). 

Targeted Competency Standard 

Evidence That Standard Was Met 

Method of Collecting Evidence 

Person (Evaluator) Responsible 

Month 

Taught 

Month 

To Be Evaluated 
Percentage of Student Mastery 



It was determined that the project evaluator would have access to the 
reported results, and each school would have access only to its own results. 
Schools were encouraged to use the results for guidance improvement efforts. 
Because the comprehensive guidance program is in place in all schools, results 
were reported as aggregate measures reflecting the districtwide guidance and 
counseling program. Aggregate results were reported throughout the district 
and summarized in reports to the board of education. This information fostered 
the sense of teamwork and enthusiasm of K— 12 school counselors throughout 
the district. 
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In addition, the school district and community became aware of the 
progress being made toward a well-defined comprehensive school guidance 
program. Districtwide support was enhanced by program results and took the 
forms of positive recognition, authorization for additional counselors, and an 
increase in financial resources for clerical assistance and materials. Although 
the evaluation process continues to be refined, district personnel and community 
members seemed impressed that counselors were attempting to evaluate and 
demonstrate how their program influenced student learning and behavior. 

Two counselors were asked to work during the summer to review reported 
results from the schools and identify more accurate and efficient ways of 
assessing targeted competencies and other components of the guidance program. 
Although tremendously time consuming, revising the assessment methods and 
clarifying data gathering procedures results in an improved evaluation process 
each year (Molyneaux, 1999). One issue identified was that for effective results- 
based evaluation to occur, the guidance curriculum competencies must be first 
written with evaluation in mind. 



Recommendations for Pursuing Results-Based Evaluation 

Arriving at a practical means of conducting results-based evaluation 
remains a|remendous challenge for school counselors. Isolating variables 
in order to obtain valid results that reflect the counselors’ contributions 
is an extremely challenging task (Molyneaux, 1999). In addition, the 
realities and pressures of working in schools make time a limiting factor. 

Enlisting the support of university counselor education faculty in 
addressing problems and providing insights can add much needed research 
expertise during the planning stages of the evaluation process. In addition, this 
experience of consulting with school counselors may encourage counselor 
educators to their counselor trainees teach practical research skills and methods 
applicable to results-based evaluation of a comprehensive competency-based 
guidance program (Paisley & Hayes, 1997). In their professional preparation, 
it is important for school counselors to leam the value of collaborative action 
research (Pine, 1981). Learning appropriate uses of surveys, qualitative 
questionnaires, case studies, needs assessments, pre- and post- tests, and program 
audits is of practical value given the difficulties of conducting research in K— 
12 schools (Allen, 1992). 

Other lessons learned in the transition toward results-based evaluation 
include these suggestions: 

• Prioritize and measure each target competency as objectively as 
possible. 

• Develop forms that will drive the assessment process and guide the 
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gathering of data. 

• Minimize the amount of time that counselors will need to devote to 
evaluation. 

• Develop a school counselor appraisal guide that corresponds with the 
comprehensive guidance program. 

• Join and communicate with other school districts that are attempting 
to define and report results. 

• It is essential to ask who, what, when, and how questions: What is to 
be evaluated? When is the evaluation to be done? How is the evaluation 
to be completed? How are data to be tabulated and reported? How can 
school counselors ensure that they do not feel threatened by the 
evaluation process? Who is to participate in the evaluation process? 
Keeping counselors, administrators, and advisory committees well 

informed of the progress being made and the barriers to overcome is 
essential. 

Equally important are constant reminders of the primary purpose of 
results-based evaluation: to “provide evidence as to how students are 
different’ as a result of school counselors delivering a comprehensive 
guidance program” (Molyneaux, 1999). v 

• We encourage guidance supervisors and other administrators to make 
comipittee work fun. Without an element of laughter, a good dose of 
praise for the counselors who are willing to take risks with a pioneering 
spirit, nourishing refreshments, a convenient meeting time, and the 
establishment of a sense of teamwork, counselors could easily become 
discouraged over the difficulty of such monumental work. 

• Ultimate responsibility for the comprehensive guidance program and 
the evolving system for districtwide delivery and evaluation rests with 
the guidance program leader (Henderson & Gysbers, 1998). 

It is not by chance that the Omaha Public Schools have received national 
recognition for having a well-defined and successful comprehensive 
competency-based guidance program. Much thought and insight went into 
developing a comprehensive competency-based program, creative program 
implementation occurred, and presently efforts to determine the results of 
delivering the program are underway. As a direct result of collaborations between 
counselors, administrators, and community members, we demonstrated that a 
team approach pays dividends for students. 

Determining how students are different’ as a result of school counselors 
delivering a comprehensive competency-based guidance program” remains a 
challenge. However, with each passing year new methodology has been 
discovered and generated to make our results-based evaluation increasingly 
practical, valid, and reliable. Evaluation is a perpetual professional process that 
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requires taking risks, persevering, and finding answers along the journey but 
benefits students and engenders a pride in. our profession. " 
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